Skip to content

feat(network): support additional destination CAs - #3292

Open
jhjaggars wants to merge 10 commits into
NVIDIA:mainfrom
jhjaggars:feat/network-supervisor-additional-ca
Open

jhjaggars wants to merge 10 commits into
NVIDIA:mainfrom
jhjaggars:feat/network-supervisor-additional-ca

Conversation

@jhjaggars

@jhjaggars jhjaggars commented Sep 11, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Add a global network-supervisor configuration for additional destination CA certificates so sandbox egress can trust private PKI without replacing default roots or changing gateway control-plane trust.

This series adopts the mechanism merged to main in #3996 (from #3781) and extends it to every first-party compute driver: one operator pre-assembled PEM bundle, the --additional-ca-bundle supervisor flag, and the /etc/openshell/tls/additional/ca-bundle.pem staging path.

Related Issue

Fixes #3295

Changes

  • Add [openshell.supervisor.network].additional_ca_bundle (single pre-assembled PEM file) with strict, bounded startup validation and fail-closed capability-based driver handling.
  • Keep effective-config preflight side-effect-free; stage gateway-owned trust material only after the selected driver passes compatibility checks.
  • Extend the shared network supervisor with additive rustls and child-process trust while preserving user TLS variables in direct mode and keeping proxy CA and gateway mTLS trust separate.
  • Stage the normalized bundle through read-only Docker/Podman mounts, Kubernetes immutable content-addressed ConfigMaps, and VM overlays, all at the upstream mount path.
  • Record and validate the expected trust generation at the gateway/driver boundary so mutable or stale runtime material fails closed.
  • Reconcile stopped sandboxes whose recorded trust generation differs from the restarted gateway snapshot while retaining sandbox identity and supported durable workspace contents.
  • Restrict Kubernetes ConfigMap RBAC to the operations required for immutable generation creation and validation; do not grant list, watch, patch, update, or delete for this feature.
  • Enforce one deployable, bounded trust-bundle limit across gateway, Kubernetes, VM, and supervisor consumers.
  • Reject additional_ca_bundle in operator driver JSON config so the global gateway setting stays the single source; standalone drivers self-source the same bundle from --sandbox-additional-ca-bundle with identical validation.
  • Add shared cross-driver additional-CA coverage for private/public trust, hostname enforcement, invalid staged material, callback isolation, and removal/restart behavior.
  • Update gateway reference, architecture, RFC, Helm, compute-driver, and cluster-debugging documentation.

Why lifecycle reconciliation is included

Additional CA sources are startup-only gateway configuration. Updating the source file or Helm source ConfigMap does not hot-reload running supervisors. To roll certificates, an operator updates the source, restarts or redeploys the gateway, and explicitly stop/starts each affected sandbox.

Each runtime records the immutable trust generation it was created with. Without reconciliation, ordinary stop/start cannot reliably move an existing sandbox to the new generation: Docker and Podman retain old container metadata, Kubernetes retains the old content-addressed ConfigMap mount, and VM retains the old writable overlay. Replacing only the staged file would conflict with the recorded digest and correctly fail closed.

The only existing manual alternative is to delete and recreate each affected sandbox, or create a replacement sandbox and move work to it. Sandbox deletion normally removes its driver-owned mutable filesystem storage—Docker/Podman workspace volumes, the Kubernetes sandbox PVC, or the VM writable overlay. Operators would therefore need to back up and restore required /sandbox contents, use external persistent storage, and re-establish any sandbox-specific state. The reconciliation implemented here avoids that migration by rebuilding stale runtime resources during stop/start while preserving the sandbox record and supported durable workspace data.

This lifecycle work can be split into a follow-up only if destructive replacement plus explicit data migration is an acceptable interim certificate-rotation procedure.

Testing

  • mise run pre-commit passes
  • Workspace formatting and Clippy with warnings denied pass
  • Targeted Rust tests pass across core, sandbox, supervisor-network, Docker, Kubernetes, Podman, VM, server, and gateway packages
  • Helm tests pass: gateway and workspace chart suites including Rust loader validation of the rendered gateway TOML
  • Unit and process-level tests added/updated
  • Cross-driver E2E coverage added and feature combinations compile
  • Rootless Podman manually verified against a private-PKI GitLab endpoint: TLS failed without the additional CA and succeeded with it while default public trust remained enabled
  • Full mise run test task run as a single validation command
  • Docker, Kubernetes combined/sidecar, and VM E2E runtime lanes (configured for CI; not run locally)

Checklist

  • Follows Conventional Commits
  • Commits are signed off (DCO)
  • Architecture and user-facing configuration documentation updated

@copy-pr-bot

copy-pr-bot Bot commented Sep 11, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@jhjaggars

Copy link
Copy Markdown
Contributor Author
 [openshell.supervisor.network]
   additional_ca_cert_paths = [
     "/tmp/openshell-e2e-podman.McovxM/additional-ca/ca.crt",
   ]

@politerealism politerealism left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Summary

The design is sound and directly addresses both #3781 (Podman TLS-trust regression) and #3295 (general additional-destination-CA feature): trust material is scoped to the supervisor container only, root-store construction is fixed for both the bundled-ca-roots and native-store variants, validation is fail-closed end-to-end, and the gateway-mTLS trust boundary is provably untouched (new destination_ca_cannot_authenticate_gateway regression test). However, the branch as submitted (eb2db3f65) doesn't compile its own test suite, so CI can't pass yet.

Blocking

cargo test -p openshell-driver-podman -p openshell-driver-docker -p openshell-server (and mise run pre-commit's rust:lint step) fail to compile, all four attributable to this PR's own commits:

  • crates/openshell-driver-podman/src/container.rs:1833 — new test additional_destination_trust_is_delivered_only_to_supervisor_with_digest is missing the pre-existing required field rootless on IsolationSpecInput.
  • crates/openshell-driver-podman/src/container.rs:2020 — existing rootful_specs = build_isolation_specs(...) wasn't updated with the new required field network_trust.
  • crates/openshell-driver-docker/src/tests.rs:135 — sets daemon_version: "28.0.0".to_string(), but DockerDriverRuntimeConfig has no such field.
  • crates/openshell-server/src/compute/mod.rs:13576 — new test start_sandbox_request_uses_optional_wire_field_six_for_durable_snapshot is missing the pre-existing required field expected_runtime_identity on StartSandboxRequest.

openshell-driver-kubernetes, openshell-driver-vm, and openshell-gateway build and test cleanly in isolation, so this is localized to four struct literals, not systemic.

Non-blocking notes

  • deploy/helm/openshell/templates/clusterrole.yaml:82-97 / role.yaml:65-70: RBAC grants unscoped get on ConfigMaps (in addition to create), since resourceNames can't restrict dynamically digest-named generations. That lets the gateway service account read any ConfigMap in the bound scope, not just its own CA generations. Reasoning is documented in-line and in the debug-openshell-cluster skill update, but worth a second look given it's a real (if bounded) privilege widening.
  • crates/openshell-supervisor-network/src/run.rs:212-219: the additional-CA load path fails closed correctly but doesn't emit a ConfigStateChangeBuilder OCSF event the way the adjacent TLS-termination code does at run.rs:401-429. Not required by AGENTS.md's OCSF scope (that's openshell-sandbox-specific), but would be a nice consistency improvement for operator visibility.

Testing

cargo fmt --all -- --check passes. cargo test -p openshell-supervisor-network -p openshell-core -p openshell-supervisor passes (1344/1345, one pre-existing unrelated flake). Docker/Kubernetes/VM e2e lanes haven't been run (not available in my review environment) and remain genuinely unverified — recommend running those before merge, in addition to fixing the compile errors above.

@jhjaggars
jhjaggars force-pushed the feat/network-supervisor-additional-ca branch from eb2db3f to 60826fb Compare September 29, 2026 13:21
politerealism added a commit to politerealism/OpenShell that referenced this pull request Oct 13, 2026
Since v0.1.2 the Podman supervisor runs as its own container, separate
from the workload image, so it can no longer inherit a private CA
baked into the workload's system trust store. Policy-allowed,
inspected HTTPS requests to destinations signed by that CA fail
upstream TLS establishment, and the only workaround is rebuilding the
supervisor image with the CA baked in.

Add a Podman-only additional_ca_bundle setting, independent of the
existing proxy_ca_bundle (which requires https_proxy), that bind-mounts
an operator PEM into the supervisor container and folds it into the
upstream trust store alongside the system CA bundle. Validated on the
gateway host at sandbox-create time, matching the existing corporate
proxy CA bundle's validation, so a misconfigured path or
certificate-free bundle fails closed immediately rather than producing
a failed sandbox once the supervisor container starts.

Scoped to Podman only, reusing the existing additive-bundle mechanism
without new gateway.toml schema, proto fields, or cross-driver
plumbing -- a narrower alternative to the in-flight cross-driver
implementation in NVIDIA#3292, which remains the intended long-term
direction.

Fixes NVIDIA#3781.

Signed-off-by: politerealism <burdcat17@gmail.com>
@github-actions

Copy link
Copy Markdown

This pull request has had no activity for 14 days and is now marked stale.

@github-actions github-actions Bot added the state:stale Inactive item at risk of automatic closure. label Oct 14, 2026
@jhjaggars
jhjaggars force-pushed the feat/network-supervisor-additional-ca branch from 5672597 to 309301d Compare October 15, 2026 20:51
opensource-fork-sync Bot pushed a commit to cla7aye15I4nd/opensource-NVIDIA__OpenShell that referenced this pull request Oct 15, 2026
…ss (NVIDIA#3996)

* fix(podman): trust an operator-supplied additional CA for direct egress

Since v0.1.2 the Podman supervisor runs as its own container, separate
from the workload image, so it can no longer inherit a private CA
baked into the workload's system trust store. Policy-allowed,
inspected HTTPS requests to destinations signed by that CA fail
upstream TLS establishment, and the only workaround is rebuilding the
supervisor image with the CA baked in.

Add a Podman-only additional_ca_bundle setting, independent of the
existing proxy_ca_bundle (which requires https_proxy), that bind-mounts
an operator PEM into the supervisor container and folds it into the
upstream trust store alongside the system CA bundle. Validated on the
gateway host at sandbox-create time, matching the existing corporate
proxy CA bundle's validation, so a misconfigured path or
certificate-free bundle fails closed immediately rather than producing
a failed sandbox once the supervisor container starts.

Scoped to Podman only, reusing the existing additive-bundle mechanism
without new gateway.toml schema, proto fields, or cross-driver
plumbing -- a narrower alternative to the in-flight cross-driver
implementation in NVIDIA#3292, which remains the intended long-term
direction.

Fixes NVIDIA#3781.

Signed-off-by: politerealism <burdcat17@gmail.com>

* fix(podman): keep additional CA mount on the supervisor

build_isolation_specs() keeps only trusted_mount() destinations on the
supervisor, and the additional CA path was not in that list. The bind
mount stayed on the workload while the supervisor received
--additional-ca-bundle for a file it did not have, so startup failed.

Add the path to trusted_mount() and cover the final supervisor and
workload specs with a test. Correct the docs and a code comment to say
the bundle is validated when the driver starts, and add Podman
troubleshooting guidance to the debug-openshell-cluster skill.

Signed-off-by: politerealism <burdcat17@gmail.com>

---------

Signed-off-by: politerealism <burdcat17@gmail.com>
@github-actions github-actions Bot removed the state:stale Inactive item at risk of automatic closure. label Oct 16, 2026
@jhjaggars
jhjaggars force-pushed the feat/network-supervisor-additional-ca branch from 309301d to aaf328b Compare October 16, 2026 18:36
@jhjaggars
jhjaggars force-pushed the feat/network-supervisor-additional-ca branch 2 times, most recently from 78d3eb8 to e94faaa Compare October 16, 2026 21:07
@TaylorMutch

Copy link
Copy Markdown
Collaborator

gator-agent

Blocked

Issue #3295 now has state:accepted; I checked the linked issue and confirmed that the failed acceptance run predates that update. PR #3292 is open, ready for review, mergeable, and passing DCO, but Branch Checks and Helm Lint are waiting for the current-head test mirror. Independent Gator code review has not run because this process gate remains blocked.

Action required: a maintainer must rerun PR Issue Acceptance, apply test:e2e for the network-trust and cross-driver lifecycle changes, and post /ok to test e94faaa35d251edb5bdb2bab9e9adf1ae54120fd as a separate command-only comment. Follow the E2E Label Help instructions and confirm the required workflows actually start. The operator has reserved test labels and /ok to test for a maintainer.

No test dispatch, approval, merge, or contributor-branch changes were performed.

Gator metadata
  • Head SHA: e94faaa35d251edb5bdb2bab9e9adf1ae54120fd
  • Base SHA: 35a3390dbc03112805cf201bb3fd99f4836a77ae
  • Merge base SHA: 35a3390dbc03112805cf201bb3fd99f4836a77ae
  • Patch ID: d5a7742b728213c34490bcfdcff56f494360d827
  • Gator payload: 12
  • Code review: not started; process gate blocked
  • Next state: gator:blocked
  • Blocked reason: test_dispatch_required

@TaylorMutch TaylorMutch added the gator:blocked Gator is blocked by process or repository gates label Oct 16, 2026
@mrunalp

mrunalp commented Oct 16, 2026

Copy link
Copy Markdown
Collaborator

Thanks for this. Private-PKI egress is a real gap, and the core trust handling is careful: the bounded single read, re-encoded PEM, a content-addressed artifact that is never overwritten, drivers having to opt in, and destination trust kept separate from gateway mTLS. The concerns below are mostly about the stop/start rebuild ("reconciliation") work and upgrades.

Blocking

1. Upgrading puts running Docker/Podman sandboxes into Error, even with no CA configured

When the gateway restarts, recovery calls start_sandbox for sandboxes in a running phase. Before this PR, the driver treated an already-running container as a no-op success. Now network_trust_generation_matches() runs before that check. A missing label counts as a mismatch, and every container created before this PR has no label. So the driver returns refusing to replace a running … workload, and recovery marks the sandbox StartFailed.

  • Podman: crates/openshell-driver-podman/src/driver.rs ~1837-1843
  • Docker: crates/openshell-driver-docker/src/lib.rs ~2735-2740
  • Recovery error path: crates/openshell-server/src/compute/mod.rs ~3884

Rotating the CA hits the same path: change the bundle, restart the gateway, and every running sandbox goes to Error. That contradicts the documented flow, where running supervisors keep their startup trust and operators stop/start each sandbox to pick up the new bundle.

2. On upgrade, every stopped Docker/Podman sandbox is rebuilt, which can lose data or block start

Because a missing label is never treated as none, the first start after upgrade rebuilds each stopped sandbox, even on gateways with no CA configured:

  • Docker copies only workspace_root into the replacement (copy_docker_workspace_archive, ~4876). Anything else changed in the container's filesystem is lost, such as installed packages or files in home outside the workspace.
  • That archive copy rejects any entry that isn't a regular file or directory, so symlinks and hard links are refused. Those are common in Python venvs and node_modules/.bin. It also caps the copy at MAX_WORKSPACE_ARCHIVE_BYTES (512 MiB). Affected sandboxes fail to start, every time, where stop/start used to work.
  • Podman: please check images whose WorkingDir isn't /sandbox. As far as I can tell they get no managed volume and no archive copy, so their workspace isn't carried over.
  • The image is re-resolved from the stored snapshot, so a tag that has moved will silently upgrade the image.

The Docker rebuild also seems unnecessary. Docker already re-spawns the supervisor from current config on every start (spawn_docker_control_process, ~2301), and the CA is never mounted into the workload, so the rebuild seems to change only the label. For Podman, recreating just the supervisor container should be enough.

Suggestion: treat a missing label as none, and/or only rebuild the supervisor side.

3. Recovery now marks sandboxes Error for config problems unrelated to their record

crates/openshell-server/src/compute/mod.rs ~3750 (also ~2198, ~4037): when driver_start_request fails during recovery, the code now calls persist_invalid_start_snapshot. On main, recovery logged a warning and skipped the sandbox. driver_start_request also fails for gateway configuration problems, for example validate_launch_signer_configured when an isolation backend is configured but the JWT signer is missing.

With this PR, those sandboxes go to Error/InvalidPersistedSandbox, with no !recoverable_error guard like the neighbouring error paths have. That reason isn't covered by is_recoverable_error_reason, so the sandboxes are skipped on every later restart, even after the operator fixes the config. The error message, which says the persisted driver snapshot is invalid, also points at the wrong cause. The test change from start_persisted_sandboxes() to the authenticated variant (~17243) seems to reflect this behaviour change.

Should fix

  • Migration from fix(podman): trust an operator-supplied additional CA for direct egress #3996 (v0.1.5-pre.4):
    • [openshell.drivers.podman] additional_ca_bundle now fails with a plain serde "unknown field" error (#[serde(skip)] combined with deny_unknown_fields). A targeted message pointing to [openshell.supervisor.network] would help. The current rejection test passes an array, so it fails on the type and doesn't cover this case.
    • docs/how-it-works/sandboxes/runtimes.mdx:205 still documents the Podman driver option, and still says the workload trusts the CA.
  • Standalone Podman --sandbox-additional-ca-bundle: operator_network_trust checks the operator's file directly, but verify_artifact requires mode exactly 0444. A typical 0644 CA file lets the driver start, then every create and start fails. Separately, the standalone path hashes the raw file bytes, while the gateway hashes the re-encoded bundle, so the same CA file gives two different generations.
  • Podman rollback on failure (please check):
    • The cleanup paths (driver.rs ~1300-1346) look like they delete the per-sandbox TLS/proxy-auth secrets even in reuse mode. The restored original pair would then fail to start.
    • An error after both renames, for example from secret_exists(...)? at ~1745, or a failed final remove_container_preserving_volumes (~1791), would leave *-rollback-* containers that delete_sandbox never cleans up.
  • Kubernetes ConfigMap validation: the exact label comparison (driver.rs:186) rejects ConfigMaps whose labels were added by mutating admission webhooks, such as Kyverno or Gatekeeper mutation. The ConfigMap is immutable and the gateway can't delete it, so sandbox creates in that namespace fail until an admin deletes it by hand. Checking that our owned labels are present (a subset match) would avoid this.
  • Helm: _toml.tpl ~312 rejects gatewayConfig.openshell.supervisor.network only when additionalCaConfigMapName is set. Setting additional_ca_bundle through gatewayConfig directly renders, but no ConfigMap RBAC is granted, so Kubernetes sandbox creates fail with 403.
  • CI: the new Kubernetes additional-CA row pins agent_sandbox_version: v0.5.0 (branch-e2e.yml:436). That looks like a rebase leftover, since main uses v1.0.3.

Smaller items / nits

  • sandbox_runtime.rs:385 ignores the digest (_digest). Either pass it to the supervisor and verify it there, or drop the parameter and the pre-ungate re-checks. As written, they don't close the window before kubelet mounts the ConfigMap.
  • CA ConfigMaps are never deleted (no ownerReferences, no delete verb). Each rotation leaves one more per namespace. The comment at ~165 about stopped PVCs needing old material doesn't seem to apply, since resume always mounts the current bundle.
  • The PEM structure check accepts any upper-case label, and rustls then silently skips unknown blocks such as TRUSTED CERTIFICATE. This fails closed, but contradicts the "no valid subset is accepted" doc comment.
  • e2e:
    • The public-trust check depends on https://example.com/ (additional_ca.rs:885), so it isn't hermetic.
    • The hostname check likely fails on the supervisor's intercept certificate rather than on hostname validation against the private CA.
    • The removal check accepts any non-zero curl exit.
    • 172.0.0.0/8 in allowed_ips was probably meant to be 172.16.0.0/12.
  • The Docker additional-ca row is added to the default matrix, so it also runs in release-dev and release-qualification. Is that intended?
  • skills/debug-openshell-cluster/SKILL.md ~542-547 names the wrong annotation (the code uses openshell.ai/additional-ca-generation). It also describes a supervisor digest argument and a network-init re-check that don't exist.
  • rfc/0003-gateway-configuration/README.md still says additional_ca_cert_paths (a list), but the implementation is additional_ca_bundle (a single path).
  • Unused or duplicated code:
    • TlsEnvironmentMode::FillMissing and its reassert loop have no caller.
    • normalized_pem() duplicates pem().
    • network_trust and network_trust_configured overlap.
    • The VM driver re-implements PEM normalization even though the digest already matches.
  • openshell-core adds a direct libc dependency, not pinned in the workspace, just for open flags that rustix/nix already provide.
  • The configuration.mdx section is heavy on internals (generations, content addressing, RBAC), and repeats material from the Helm README and the skill.
  • The PR body says feat: trust additional destination CAs for sandbox egress TLS #3295 is awaiting state:accepted. It's accepted now.

Overall

The PR body offers to split out the lifecycle reconciliation, and I'd take that offer. Most of the risk above comes from the stop/start rebuild machinery, not from CA delivery. A smaller first PR could do CA delivery and validation, plus "stop/start picks up the current bundle" by recreating only the supervisor, as the Kubernetes driver already does. That would land the feature with far less upgrade risk. Rebuilds that preserve the workspace could follow, with their own upgrade tests.

@jhjaggars
jhjaggars force-pushed the feat/network-supervisor-additional-ca branch from e94faaa to 7154342 Compare October 16, 2026 22:17
@TaylorMutch

Copy link
Copy Markdown
Collaborator

gator-agent

Blocked

Issue #3295 is accepted, and PR Issue Acceptance and DCO now pass on the latest head. Branch Checks and Helm Lint are waiting for the current-head test mirror. The green E2E status means test:e2e is absent, not that E2E ran. Independent Gator code review has not started because the test-dispatch process gate remains blocked.

Thanks @mrunalp for identifying the upgrade and stop/start reconciliation risks. I verified your maintainer authority and checked the latest head, review history, and CI dispatch state. Your implementation feedback remains input for the eventual code review; this process check does not establish whether the latest changes address it.

Action required: a maintainer must apply test:e2e for network trust and cross-driver lifecycle coverage, then post /ok to test 7154342135224d7a841cb00d8068feabc4d0563b as a separate command-only comment. Follow the E2E Label Help instructions, approve any required workflow execution, and confirm Branch Checks, Helm Lint, Trivy Changes, and required E2E workflows actually start on the current head. Acceptance no longer needs a rerun.

Gator metadata
  • Head SHA: 7154342135224d7a841cb00d8068feabc4d0563b
  • Base SHA: 604b9f66f685c582114cd571ae3a10accec23ec2
  • Merge base SHA: 604b9f66f685c582114cd571ae3a10accec23ec2
  • Patch ID: 5dafe68e34718f726af916a4e0c75d323de06902
  • Gator payload: 12
  • Code review: not started; process gate blocked
  • Next state: gator:blocked
  • Blocked reason: test_dispatch_required

@TaylorMutch TaylorMutch added the test:e2e Requires end-to-end coverage label Oct 16, 2026
@github-actions

Copy link
Copy Markdown

Label test:e2e applied, but pull-request/3292 does not exist yet. A maintainer needs to comment /ok to test 7154342135224d7a841cb00d8068feabc4d0563b to mirror this PR. Once the mirror exists, re-apply the label or re-run Branch E2E Checks from the Actions tab.

@TaylorMutch

Copy link
Copy Markdown
Collaborator

/ok to test 7154342

@github-actions

Copy link
Copy Markdown

Compatibility review (advisory)

New or updated breaking changes; see the full report for all findings.

5. rust-sdk — No explicit risk acceptance found

openshell_sdk::raw::proto::compute::v1::StartSandboxRequest gains a new generated sandbox field.

  • Impact: Raw Rust SDK clients, compute-driver mocks, and exhaustive patterns using the generated struct no longer compile until they supply or ignore the new field.
  • Action: Remove the generated-field change from the patch train, or release it in a minor version with guidance to set sandbox: None or use ..Default::default().
  • PR: feat(network): support additional destination CAs #3292
Review details
  • Execution: complete
  • Candidate: PR #3292 (44bf35384753a92d96fcafdac390e6c87c86c4a7)
  • Baseline: 604b9f66f685c582114cd571ae3a10accec23ec2 (604b9f66f685c582114cd571ae3a10accec23ec2)
  • Reviewed: rust-sdk, python-sdk, go-sdk, typescript-sdk, api-behavior, cli, configuration, policy, helm, state

Advisory review; publication gates are unchanged.

Branch check run

@jhjaggars
jhjaggars force-pushed the feat/network-supervisor-additional-ca branch 3 times, most recently from eb4a379 to 94b285d Compare October 17, 2026 01:44
Signed-off-by: Jesse Jaggars <jjaggars@redhat.com>
Signed-off-by: Jesse Jaggars <jjaggars@redhat.com>
Signed-off-by: Jesse Jaggars <jjaggars@redhat.com>
Signed-off-by: Jesse Jaggars <jjaggars@redhat.com>
Signed-off-by: Jesse Jaggars <jjaggars@redhat.com>
Signed-off-by: Jesse Jaggars <jjaggars@redhat.com>
Resolve rebase fallout against current upstream main after the series was
rebased 214 commits forward:

- follow upstream NVIDIA#3110: supervisor→gateway TLS is CA-only, so
  configure_gateway_tls and its isolation tests drop the client identity
- follow upstream tls_secret_names reduction to [String; 1] in the Podman
  driver while keeping the network trust bundle delivery
- port [openshell.supervisor.network] rendering into the
  openshell.gatewayConfigToml serializer in _toml.tpl
- adapt tests to upstream struct/function changes (DriverStartupContext,
  SandboxRunConfig, IsolationSpecInput, supervisor_pod, VM bootstrap image)
- drop conformance-artifact-prefix plumbing removed upstream

Signed-off-by: Jesse Jaggars <jjaggars@redhat.com>
Unify the additional destination CA series on the mechanism merged to
main in NVIDIA#3996: one operator pre-assembled PEM bundle, the
--additional-ca-bundle supervisor flag, and the
/etc/openshell/tls/additional/ca-bundle.pem staging path.

- Replace the gateway-level additional_ca_cert_paths list with a single
  additional_ca_bundle file validated and staged at gateway startup.
- Restore the upstream supervisor argument path and fold-into-roots
  behavior; drop the --network-additional-ca-bundle and
  --network-additional-ca-digest contracts.
- Keep gateway normalization bounds, content-addressed staging,
  digest-based driver reconciliation, and stop/start rotation; the
  freshness check now runs at the gateway/driver boundary instead of
  the supervisor boundary.
- Reject additional_ca_bundle in operator driver JSON config; standalone
  drivers self-source the bundle from --sandbox-additional-ca-bundle
  with the same validation.
- Extend the Kubernetes ConfigMap name digest segment to keep the full
  63-byte budget after the prefix rename.
- Box the persisted-sandbox reconciliation futures so callers stay
  under the clippy::large_futures threshold.

Updates Docker, Podman, Kubernetes, and VM delivery, Helm chart and
tests, e2e wiring, gateway configuration docs, and the cluster
debugging skill to the single mechanism.

Signed-off-by: Jesse Jaggars <jjaggars@redhat.com>
… lockfile

The network trust startup tests asserted on raw substrings of gateway
diagnostics, but the CLI renderer wraps output at a fixed width with
continuation markers and hard-breaks long paths. Compare assertions on
whitespace- and continuation-squashed text so they hold regardless of
wrap position.

Format the e2e workspace (separate cargo workspace, not covered by the
root cargo fmt run) and refresh examples/isolation-backend-v8/Cargo.lock
for the libc dependency added to the network trust crate.

Signed-off-by: Jesse Jaggars <jjaggars@redhat.com>
The additional CA e2e never ran in CI before the first maintainer test
dispatch, and its local container inspection was broken in two ways: it
located the supervisor container through e2e network membership (the
supervisor is host-networked or runtime-bridged, so the filter matched
the CI job container instead), and it required the gateway mTLS CA as a
bind mount (Podman delivers it as a container secret when user
namespaces remap UIDs). Select the supervisor by managed labels and
accept the secret delivery form.

The lifecycle leg then exposed a driver bug: stopped-pair trust
replacement re-created the per-sandbox TLS CA secret, colliding with
the secret still owned by the renamed stopped pair. Mirror the token
and resolver secret handling: require and reuse the existing secret
during replacement instead of creating it.

Validated end-to-end with the Podman additional CA e2e lane.

Signed-off-by: Jesse Jaggars <jjaggars@redhat.com>
@jhjaggars
jhjaggars force-pushed the feat/network-supervisor-additional-ca branch from 94b285d to 15f5d86 Compare October 17, 2026 01:59

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

gator:blocked Gator is blocked by process or repository gates test:e2e Requires end-to-end coverage

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat: trust additional destination CAs for sandbox egress TLS

4 participants